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ABSTRACT 

The purpose of this paper is to encourage the reader 
to distinguish between evaluation and other i tf criaticn-gathering 

procedures, to distinguish between Measures of efficiency and 
effectiveness, and to pake inioried decisions about when to useN 
evaluation rather than another procedure for producing infornation 
about a project. Material is divided into types of 
iaforiaticn-gathexing activities, types of evaluative research, 
preliiinary questions preceding selection cf type of evaluation, 
suggestions for further resources relative to the technical aspects 
of evaluative research and an outline of tasks involved. Although 
specifically written for judicial programs for juveniles, these 
step*by-step procedures can b€ applied to other evaluation efforts. 
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INTRODUCTION 



A good evaluation presents many difficulties. Designing and implement- 
ing a good piece of evaluation requires staff cooperation: \^ 

« Staff must be willing to accept that rigorous standards will 
be appilied in the evaluation. 

« Staff must be prepared for unanticipated setbacks in the 
research tasks accompanying an evaluation effort. 



Staff must understand that findings will demand careful and 
thoughtful interpretation. 

Staff must honestly decide whether or not the results warrant 
the continuation of an evaluation effort. 



Evaluation should be. considered an appropriate activity worth the effort, 
rather than a casual project taken up for unclear or unspecifiable 
reasons. Unless it is regarded as a choice, there are other types of 
documentation, rejxjrting, and analyses that could be applied to a proj- 
ect and that may be more appropriate. 

■ The purpose of this document is to encourage the reader to distinguish 
between evaluation and other information-gathering procedures, to dis- 
tinguish between measures of efficiency and effectiveness, and to make 
more informed decisions about when to use evaluation rather than some 
other procedure for producing information about a project. This document 
is not an attempt to detail how evaluatidns are- designed and implemented. 
There are good resources already available for this which would help an 
agency or project staff to determine whether it has the necessary staff, 
or would suggest what sort of assistance the staff will want to seek out- 
side the organization. 

The material is divided into: 

• Xj^es of information-gathering activities. 

• Types of evaluative research. 

A list of preliminary questions to be asked before 
selecting evaluation. 

• Suggestions for further resources into the technical 
aspects of evaluative research. 

• A brief outline of the types of technical tasks that 
are involved in evaluation research. 
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TYPES OF INPOPMATION--GATHERING 

Three general but distinct procedures are commonly used: assessment, 
monitoring, and evaluation. 

Assessment is establishing the rate or amount of some activity or 
resource. Assessment of needs is a means of estimating or determining 
the amount or importance of unmet needs and identifying the resources 
currently ^available to meet those needs. Needs assessment sometimes 
uses existing data or involves collection of nev data. It is a method 
of finding Service delivery gaps and substantiating unmet needs in a 
comnunity and is used to establish priorities for addressing problems. 

Monitorincj coii?>ares a project's plans with what actually happened. It 
entails collecting specific information on events associated with the 
operation of a project. In general, a monitoring system obtains data 
on both the project and its activities, allows for the analysis neces- 
sary to determine whether activities are acceptable, and provides for 
feeding back this information to management. Monitoring activities are 
associated with reporting systems amd cost analysis techniques. 
Specific information on this function is presented in Monitoring for 
Criminal Justice Planning Agencies (LEAA, 1974) . A more general reference 
is Monitoring for Government Agencies by John Waller, et al. (The 
Urban Institute: Washington, D.C. 1976). 

Evaluation. refers to judging the merit of something by comparing it 
against some yardstick. Evaluation studies are done to measure the 
effects of a program or project against the objectives it set out to 
accomplish and thus aid subsequent decisions about the project's future 
or structure. In the strictest sense, evaluation studies using systematic 
methods are designed to verify whether a certain effect occurred and to 
suggest conclusions about the extent to which this effect can be directly 
attributed to i/ne project rather tham to outside forces. 



TYPES OF EVALUATIVE RESEARCH 

r 

As evaluation research has become widely used in -government agencies 
and Action projects, attempts have been made to distinguish degrees 
of success or failure. Programs are designed to produce a specific 
outcome. These results are related to program input and include its 
purposes, principles, methods, staffing, location, auspices, and 
numbers and characteristics of persons served. Both input and outcome 
can be evaluated. (Suchmam, .1967) , Evaluation Research has come to 
describe systematic measurement and analysis of the effects of various 
aspects of program review. Evaluati\, research is involved when activi- 
ties are judged, against iitqplicit criteria, even if they are not judged 
against the actual outcome of program activities. 

» 

The following are examples of areas to which evalxiative research might 
'well be applied: 

• Effort . This refers to both quantity and quality of activity. 
There is the ass\imption"that the specific activity is a valid 
means of reaching the objectives, so it is measured regard-, 
less of 'output. Monitoring activities usiially cover at least 
quaritity of activity. 

#^^ Efficiency . The evaluation of alternative methods of achiev- 
ing objectives in terms of costs involved — money, time, 
*^ personnel, and public convenience ~ focuses on the efficiency 

with which procedures arie carried out. It is often done as 
cost benefit or cost effectiveness analysis. 

• Operation . The focus here Is on detailed questions concerning 
why a program works or does not. A description of operation 
would include program attributes, program recipients, program 
conditions, and details about differences or chamges in these 
and their effect on the success or failure of the program. 

• Effectiveness of Performance . This is the measure of results 
of effort, rather than of effort itself. The key question, con- 
cerns the extent to which program objectives have actually been 
attained. Methodological considerations and controlled analysis 
are needed if the study must also demonstrate that what occurred 
was a result of project activities. 

• Adequacy of Performance . This compares performance to the amount 
of need: How much of the entire problem has been solved as a# 
result of the program or project? Effectiveness measures impact 
on a population actually served ; adequacy measures impact on a 
larger population of need. 



These five categories combine to form two basic types of evaluative 
research: measures of the inputs or processes in the project ^^nd 
• measures of 'the outcome or impact of the project. While each provides 
a distinct type of information, used together they provide a balanced 



This is a detailed study of the various parts of the system and how 
they are linked. Such a study can be acconplished through careful and 
systematic documentation of staff activities, project attributes, recipi- 
ents, etc., and the conditions and changes in. any of these. Carefully 
designed reporting systems should provide for all of these things, as 
well as for adeqxaate information to make estimates, using cost analysis 
techniques, of the efficiency of the activities and services. This 
involves more than monitoring project activities and careful documenta- 
tion of project characteristics. It also requires judging the quality, 
adequacy, or appropriateness of the procedures and making necessary 
adjustments. There mist be some rationale for analyzing these aspects 
,of a project. For example, keeping records of how many staff hours were 
spent talking with clients one-to-one and how many hours were spent talk 
ing to clients in groups has a reason if the program hopes to find out 
if one-to-one contact is more or less valuable. 



The concern here is with the relationship .of project outcomes to stated 
goals. It is assumed that a problem has been identified, that there 
is a theory about what will alleviate it, that the project has a goal, 
and finally, that the program activities will have a specific impact 
on the target group. ^ 

An impact evaluation depends on a statement of the theory of what will 
alleviate the problem. It detv rmines how the project is set up and 
allows for measurement of tl'. , ^oposed effect. At a minimum, an impact 
evaluation will be a study of change in some target group (individuals, 
a section of a community, a jurisdiction, etc.). 

However, it is even more in^rtant that the study be designed for 
conclusions about the extent to which the projejt activities themselves 
created the change. For example, to say that a job program for youth 
reduced their delinquency behavior, it would be necessary to have a 
"control" or comparison group of similar youth who were not provided 
jobs. Both the group provided jobs and the control group would be 
measured by their delinquent behavior before and after the job 
program. If delinquency were lower for those given jobs than for the 
control group, it would be more persuasive evidence of the impact of 
the job program than a before and after measure on only the working 
group. 




Process Evaluation 



Impact Evaluation 



Although the sophisticated design and rigorous technique used to link 
project activity and effect on its target provides the best type of 
information in deciding 2d>out program design and existence impact 
evaluation is not always appropriate or possible. It may not be feasible 
because of limited resources, limited expertise, or inadequate program 
design. Or it may not be desirable because no one intends to act on 
that type of knowledge. 

Process and impagt evaluatj^ons are related . Fpr planning purposes, 
knowing ^at a project had a certain effect may not be as useful as 
knowing that it had an effect and why it had that effect. Using the 
job program example again, it may be found that the impact on delinquency 
was uneven atirods the group getting jobe. With adequate information on 
how the jobs for youth were created and what those jobs entailed, it may 
be possible to determine that different types of jobs were viewed dif- 
ferently and thus had different impacts on reducing delinquency. For 
example, jobs that the employer created and financed may have been viewed 
more favorably by youth than those jobs created by employers simply be- 
cause the job program was providing the salary with no real commitment 
made by the employer. 

The LEAA guidelines call for State Planning Agencies to make "intensive 
evaluations" when planning warrants it. An "intensive evaluation" 
implies a study including both process and impact evaluation which would 
facilitate adjustnvent of project acjCiivity or transfer of project design 
to others. Both process and impact must be described carefully enough so 
th^it others wishing to design a similar program for a similar impact on 
youth can provide similar staffing, service facilities and clients, or, 
if necessary, adjust aspects of proje;;t activities to produce an even 
more favorable* iinpaot. 

» 

The inf ormation gained from monitoring and docume nti ng the activities of 
a project is frequently all that is required. A" descriptive report of 
what was done at the end of a specified period may be sufficient. Even 
descriptive reports, however, require thoughtful consideration and analysis 
in order to be useful for purposes other than that of establishing compli- 
ance. There is often room, for improvement in the way activities are 
carried out, especially in the way they are planned for and incorporated 
into the project from the beginning. 

Howevo r, project staff and others may decide that- an evaluation of the 
project, particularly an evaluation of the project 'is impact, is what is 
wanted. They want to go beyond monitoring ongoing activities and deter- 
mine if ttik project activities are producing the projected outcome, and 
if this outcome is related to activities and not to other forces. The 
following section discusses preliminary steps in deciding whether or not 
such evaluation actually can and should be undertaken. There are three 
crucial questions to consider carefully before commitment to an evalua- 
tion effort: (1) Will the findings be used? (2) Is the project 
evaluable? (3) Who can do this work? 
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PRELIMINARY STEPS TO EVALUATION 



STEP i: Will the findings be used? 

When considering evaluation, first ascertain whether there is a sincere 
interest in or need for the findings, even if they are not entirely 
favorable. Who wants to use the results and for what purpose? Careful 
consideration should help to determine whether to go to the trouble and 
expense of doing an evaluation and should guide the decision as to 
whether it will be a process evaluation or an impact evaluation. Evalu- 
ation should not be perfprmed on the basis of such reasons as "it is 
required" or "it ought to be done because others do it." It should be^ 
done because the information is needed for decision making and will be 
taken into account in future planning. 

Another factor in considering whether an evaluation study actually would 
be used is that of the distinction between summative evaluations and 
formative evaluations (Scriven, 1967). 4 . 

« 

• Summative evaluations aid decisions on whether to keep or 
stop a project. 

Formative evaluations aid decisions on whether changes are 
needed in the way the program is structured or in its oper- 
ations and, if so, wh||pier or not to maUce these changes. ^ 



Agencies or communities with an experimental approach will most often be 
interested in formative evaluations: their interest will lie in improv- 
ing programs, since that is the more efficient way to plan and to develop 
a program. Formative evaluation is appropriate for feeding useful 
information back into the planning process. 

"Suitmative" and "formative" concern more the i^e of the evaluation than 
how it is conducted; but, if it is clear that formative evaluation is 
needed, process evaluation with impact evaluation* must be included. 
It is not possible to change a program's operation to improve outcomes; 
if the original processes are not understood and have not been analyzed 
with respect t<3 their results. 



STEP 2: Is the project ^valuable ? 

Once it' is decided that an evaluation is needed, the next step is to 
determine whether the project is evalyaUDle as currently cfcsigned and 
operating. The key conditions are; 

• « Ability to identify the project's objectj.ves . What is this 

project trying to acconplish? Too often this question cannot be 
answered clearly and precisely. Objectives must be stated in 



terms that are clear, specific, and measurable. The kq^ . 
assumptions of the program must bie stated in a form which 
can be tested objectively. That is, not only must the' out- 
come be definable, but also the process used to achieve it 
must be specifiable. . * 

A bility to identify the criteria for data . The final 
criteria must be specified. How. is the outcome going 
to be measctred? For example, will it be delitiguency rates 
or will it be self- reported delinquency among the clientele? 



Ability to identify the relevant groups that the evaluation 
can use to measure impacts . There must be a specifiable 
m clientele- group or population where tHe change is expected 
to occur directly. 



In determining whether tl\e project can be evaluated, it may become evi- 
dent that there are multiple objectives or multiple definitions amc the 
staff 'and that there are botlv short:-term and long-term objectives. 
E^i&luator and evaluatee must select among the objectives one or two 
which are more important or more interesting; they ^st be able tp 
agree about definitions; and they must be able to decide whether to 
measure short-term or loftg-term effects, or both. If these.- things can- 
not be agreed upon, then the use of the findings of the study is jeop- 
ardized. 



STEP 3; Who c^n do this work? 

Most of the* steps in an evaluation do not require highly specialised 
professional training as much as they require staff with a commitment 
to being rigorous, systematic, and objective. Often the agency over- 
seeing evaiuati' s or a project carrying out its own evaluation will 
have a^aff capa^.e of undertaking many parts of evaluative research. 
However, it may not mean that they also have time and this needs to be 
taken into consideration. » 

Even with Capable staff, it is still important to have someone with 
expertise and experience in this type of research to direct the tech- 
nical aspects of evaluation and to design the study and the measurement 
instrvunentS to be used. There may be such a person on the staff. If 
not, there are professional evaluators within most state government 
systems or available for hire from private firms in the community. How- 
ever, even selecting someone to hire tak'is some knowledge and understand 
iiig of the process. Agencies intending to do much evaluation or to over 
see evaluations as an ongoing activity should acquire staff with this 
capability. A checklist of what to look for in an evaluation plan is 

Appendix A. 

Another alternative is to limit the types of evaluation to those which 
the available staff can handle. Attempting a sophisticated and complex 
design without proper understanding of the techniques or skills to do 
the necessary statistical analysis and data interpretation would waste 
staff time and project resources. 



V 



Therefore, in deciding on staffing needs,- consider three things 

• What sort of evaluations will be conducted (e.g., process 

or impact, short-term or long-term, etc.) and how many 

will be conducted? > ~ ^ f 

• * 

r# ^ihat funds ar'^ available for the evaluation^'activities^? 



• What parts of thfe evaluative research could be done by . 

in-hou^e staff and v*iat parts by outside contractors or , ^ 
consultants? . ' ^ . 

Somewhat more in-depth discussions of the management of resoCiirces for 
evaluation activities are in Intensive Evaluation for Criminal Justice 
Planning Agencies (Wiedman, et al.^ 1975), Practical Program Evaluation 
for State and. Local Government Officials (Hatry, et al.,' 1973), and 
Federal Evaluation Policy (Wholey, et al,, 1976). - The first of the.^a . 
publications also gives several examples of alternative organizational 
strategies used by SPAs in the past for conducting intensive evaluation 
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TECHNICAL STEPS 



Recoiranended Resources 



The remaining steps in evaluation are technical. These are clearly 
laid out in many research texts and in government iiianuals for agency 
personnel responsiblig for conducting or overseeing evaluations. For 
exanqple, Evaluation Research; Methods of Assessing Program Effective- 
ness (Weiss, 1972) addresses in general the application of research 
techniques to the evaluation of social programs* It formulates the 
questions and various research designs and then addresses the political 
context in whl^ch- sucTTievaluation takes place* The presentation is 
cogent and easy to follow. It would be useful for an agency or prdject 
manager in evaluating staff capabilities and the time and money likely 
to be r>eeded. Other resources are: r 

^ • Practical Program Evaluation for State and Local Government ' 
Officials (Hatry, et al., 1973) is 'directed at thergovernment 
. ^ analyst. It lays out an easily followed outline of considera- 
tions and steps associated\with program evaluation. 

/ ' ' \ • • • 

• Routinizinq Evaluation; Gettinqr Feedback on Effectiveness 
of Crime and Delinquency Prlfcryams (Glaser, 1973) is a some- 
what longer narrative. Examples provided by Glaser are more 
specific to crime and delinquency than in the Weiss or Hatry 
volumes, but' the/format is Less straightforward and less 
introductory in nature. It goes >eyond impact evaluation 
and discusses jJrocedures associated, with process evaluation. 

' • Federal Evaluation Policy (Wholey, et al,, 1976) is a broa^ 
treatment of the need for a wide-ranging evalxiation strategy 
for large government programs and a discussion of administrative 
systems, organizational relationships, resp6nsibij.it ies, and 
methodology, a ^ ^ 

• Eval>uation tn Criminal Justice Programs^ Guidelines and E><aifees 
(Albright, et al., 1973) was prepared as K^ide for LEAA National 
impact Program managers • It discusses developing and implepent- 
ing plans to evaluate criminal justice projects- and programs and , 
gives examples of applying evaluation methodology and 6f effective 
ness and efficiency measures. 

• Intensive Evaluation for Criminal Justice Planning Agejt 
(Weidman, et al.) defines, according to LEAA, "intensiW^valu- 

' ation" and describes ways it can be carried out. . It -prWides 
examples of alternative organizational strategies xot ipltensive 
evaluation used by four SPAs, along with bri^f exan^fOe's of 
strategies used in states with small-sized grants aiVd of data ^ 
^ systems used by SPAs for evaluation purposes. 




Technical Tasks 
Initial tasks require a seriej of decisions: 

• What is to be measured? 

• How is it to be measursd? 

• Who, is to be measured? 

• When is it to be collected and for what groups? 

• How it is to be colle 

• How are the data to be used? 



•The reader should not be misled into thinking that selecting a design 
and carrying out research will be simple and predictable. Any of 
thS publications listed above should mak^this evident* Irt particular^ 
the Weiss and-Glaser volumes discuss not only the techniques but also 
the pitfalls associated with research efforts, especially those conducted 
in social and political contexts for use in decision making (aS compared 
to basxc research for the sake of knowledge) . 



TASK 1; Formulating the ^ >i tion 

An evaluation cannot be desifgned until : ^ogram objectives are clarified 
end stated in measurable terms. 'Pjr example, "To enhance the well- 
being of youth" is not maasnrablq; "To reduce truant behavior for youth 
in thid program" lends itself better to devising a measure of whcither or 
not this his been accomplished. If there are multiple objectives , one 
or two must be selected as more important in order to provi<ie a manage- 
able study. 2he staff itoelf and the staff and the e<^luatcr must agree 
aB2mt thi^ definition of program objectives, about the importance of one 
or two o^(^ others, about the target group to be measured, and about the 
preference of studying short-term effects, long-term effects, or both. 

Similar decisions and definitions need to be irui^^e with respect to a 
proceps evaluation. There must be a clearcut rationale for having col- 
lected certain information. This xinderlying reason guides compilation 
of data and helps judge the project^s efficiency and activities. 



TASK 2: Designing Instruments or Techniques for Measurement 

Perhaps the best strategy here is to locate suitable existing measures. 
For exaunple, carefully developed and documented scales exist for many 
psychological variables, such as alienation or self-esteem. Handbooks 
are available which describe established measures (see Lake, Miles, and 
Earle; 1973). This approach eliminates much trial and error which 
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accompanies the creation of new measuxes. It may also provide for 
responses in earlier investigations which could be used for comparison 
purposes • 

If there are no suitable measures available, then careful work must be 
done at this stage. Hastily designed instruments or techniques at the 
beginning of a study can lead to increased difficulties with making 
sense of the findings, or eveij^^ jeopardize their utility. The theory 
of measurement is highly develpped and much has been written on it. It 
would be iii^esponsible for a piroject to create measures for evaluative 
research without familiarity with the various typed and levels of measure- 
ment. Some measurements are more precise than 6thers; however, project 
design or objectives may not always allow for the most precise ones to 
be used. This needs to be examined and given careful consideration -to 
avoid aiming for a false precision. 

TASK 3; Designing the Study; Common Models for Evaluation Studies 

Bie usefulness of the findings greatly depends on this stage. If precise 
statements about the extent to which project activities are responsible 
for outcomes are most important, then sophisticated and complex designs 
must be used, and used correctly. Other designs are less complex than 
classic experimental designs to execute and analyze, but these are also 
less precise in linking effects to activities and they feave far more 
room for alternative explanations for the actual outcomes. 

There are many different models incorporating elements of experimental 
design that are appropriate to evaluative research and present a range 
of explanatory power and ease of implementation. For example. Federal 
v ;Evaluation Policy presents a chart of eleven pre-experimental, true 
exp<»rimental , and quasi-experimental designs which have been developed 
by bial scientists and discusses their relative strengths and con- 
str^iints. The classic work upon which the discussion is based is 
Experimental and Quasi-Experimental Designs for Research (Campbell and 
Stanley, 1966), written for a more technical audience. Below are four 
coninon evaluation models. These are discussed in Intensive Evaluation 
for Criminal Justice Planning Agencies and are airKjng those commonly 
described in any presentation of how to conduct program evaluation. 
P ractical Program Evaluation for State knA Local Government Officials 
describes these same four designs in somewhat more detail, addressing 
general steps involved in each one, problems likely to be encountered 
in each, types of application, and relative costs. These can serve as 
a guide for project staff considering setting up an impact evaluation. 
However, it is r<*commended that they work with someone with expertise 
and experience in setting up such studies because it is easy to view 
these designs as being simpler to implement than is actually the case. 

Designs 2, 3, and 4 on the followifig chart allow for controlling some 
pf the other factors that tnight affect the target group. Design 4, 
which is the classic controlled experiment dosign, is the most p->werful 
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in terms of being able to attribute program effects to program activities. 
It is also the most difficult to prepare, carry out, and analyze and will 
take more planning, more expertise, and more staff time than any of the 
otherj3. Design 1 is probably the most commonly used. It is the simplest 
and cheapest type of evaluation and ccn identify if there has been a 
change. But, at the same time, it is the least capable of separating 
the effect of program activities from other influences. 

A fifth model of measurement (not actually a design^ compares actual 
results against planned or targeted results. This is not an impact 
measurement because it does not measure the effects of service, but 
often cain provide adequate information for planning purpbses. It is 
generally based on such things as measures of workload and population 
served amd can be accomplished with information collected in reportihg 
systems designed for program monitoring. 

TASK 4t Data Collection 

Data collection is the most routine of the tasks, but it is likely to 
consume the greatest amount of time and effort. It takes thoughtful- 
ness and commitment on the part of the staff, for if done sloppily or 
inconsistently the legitimacy of the findings is reduced. Data must 
be accurate, complete, and comparable from time period to time period 
or from group to group. 

Though we commonly think of interviews or questionnaires as basic data 
collection methods, using existing records and statistics is probably 
more prevalent. This would be .particularly true for process evaluation 
and is by definition true when the idea of evaluation has not been 
thought of until the end of the project. Evaluators with ingenuity can 
often think of other special data collection procedures such as system- 
atic field observations, special event analysis, critical incident tech- 
niques, and so forth. These may make it more difficult or even impos- 
sible to obtain the desired "before" measures, but should not be over- 
looked if other collection methods are not applicable to the project's 
objectives or operations. 



TASK 5t Utilization of Results 

If the suggested steps are followed, staff already know how the results 
are to be used and who is going to use them, as this was a premise in 
deciding whether or not the evaluation should, be undertaken in the 
first place. 

The most important thing to understand at this point is that the data 
will not speak for themselves. The staff must be prepared to think 
about what they mean and to consider alternative explanations for 
specific findings. They should be prepared to consider the findings 
as one piece of information and not the only piece. They should be 
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DEbIGN 3 

One Time Comparison Between Those 
Served and Not Served by Program 
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Pre-Program 
Period ' i 
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Program 
Period 



NS 



Time 

Program Effect - Rate of Those Served (S) 
Rate of Those Not Served (NS) 



DESIGN 4 

Time Series Comparisons Betv^f n 
Those Served and Not Served 



Pre-Program Program 



Period 



Period 



BUi^GLARY 
RATE 




Time 



Program Effect - Difference in Before and 
After Burglary Rates of Population Served 
by the Program (S,--S,) - Similar 
Difference for Those Not Served (NS, NS.) 



Evalualion Design Methodologies 
Adapted from: Intensive Evaluation for Criminal Justice Plannin g Agencies 
(Weidman, et al., 1975) 
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prepared to determine with hpnesty whether or not the evaluation was 
carried out well enough and without too many unanticipated events in 
the environment or in the data collection procedures. They must be 
prepared to ask if the product can be used ?.s intended, if it can be 
used but only with qualifications, or if it should be scrapped. 

Despite early agreements eibout the utilization to be made of the find- 
ings, the results may prove ^'unpopular" and resistance to using them 
even for formative purposes may arise. Evaluations done for summative 
purposes may be discounted as inaccurate, even when design and data 
collection can be declared adequate. Too many evaluation studies 
simply disappear. Th.ls is the reason for deciding to use the findings 
and stating what they are to be used for and for making the first 
decision — not the final one. ^ 
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APPENDIX A 

An Administrative Checklist 
for Reviewing Evaluation Plans 



Conceptualization of Evaluation 



^Definition: 

Purpose: 
JJuestions: 
JVudiences: 

Agents 7 
JProcess: 

Standards: 



How is evaluation defined in this effort? 

What purpose (s) will it serve? 

What questions will it address? 

Who' will it serve? 

Who will do it? 

How will they do it? 

By yhat standards will their work be judged? 



Sociopolitical Factors 



Involvement: 



^Internal 
communication : 

Internal 
"credibility: 

External 
""credibility: 

Security: 



Protocol : 



Public relations: 



Whose sanction and support is required, and how 

will it be secured? 
How will, communication be maintained between the 

evaluators, the sponsors, and the system 

personnel? \ 
Will the evaluation be fair to persons l^nside 

the system? 
Will the evaluation be free of bias? 

What provisions will be made to maintain security 

of the evaluative data? 
What communication channels will be used byVthe 

evaluators and system personnel? \ 
How will the public be kept informed about the 

intents and results of the evaluation? . 



Contractual/Legal Arrangements 



j:iient/evaluator 
"relationship: 

JIvaluation 

Products : 

Delivery 

Schedule : 
^Editing: 

Access to data* 



Release of 
reports : 
^^Responsibility 
and authority : 
Finances: 



Who is the sponsor, who is the evaluator, and 
how are they related to the program to be 
evaluated? 

What evaluation outcomes are to be achieved? 

What is the schedule of evaluation services and 
products? 

Who has authority for editing evaluation reports? 
What existing data may the evaluator use, and 

what new data may he obtain? 
Who will release the reports and what audiences 

may receive them? 
Have the system personnel and evaluators agreed 

on who is to do what in the evaluation? 
What is the schedule of payments for the evaluation 

and who will provide the funds? 
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The Technical Design 



jDbjectives and 

variables: 
JLnve 3 1 igator y 

framework: 



Instrumentation : 



^Sampling* 



Data- gathering * 



JData storage 
and retrieval* 

JData analysis: 
Reporti ng : 



JTechnical 
adequacy : 



What is the program designed to achieve, in 

what terms should it be evaluated? 
Under what conditions will the data be gathered, 

e.g., experjjnental design, case study, survey, 

site review, etcr? 
What data-gathering instruments and techniques 

will be used? 
What samples will be drawn, how will they be 

drawn? 

How will the data -gathering plan be implemented, 

who will gather the data? 
What format, procedures, and facilities will be 

used to store and retrieve the data^ 
How will the data by analyzed? ^ 
What reports and techniques will be used to 

disseminate the evaluation findings? ^ 
Will the evaluative data be reliable, valid, and 

objective? 



The Management Plan 

Organizational 

mechanism: 



Organizational 
location : 



Policies and 
procedures : 
JStaff : 
Facilities : 



Jfata gathering 

schedule : 
^Reporting 

schedule : 

Training : 



Installation of 
evaluation : 

Budget : 



What organizational vinit will be employed, e.g-, 
an in-house office of evaluation, a self- 
evaluation system, a contract with an 
external agency, or a consortium-supported 
evaluation center? 

Through what channels can the evaluation 

influence policy formulation and administrative 
decision making? 

What established and/or ad hoc policies and 
procedures will govern this evaluation? 

How will the evaluation be staffed? 

What space, equipment, and materials will be 
available to support the evaluation? 

What instruments will be administered, tc what 
grotqps, according to what schedule? 

What reports will be provided, to what audiences, 
according to what schedule? 

What evaluation training will be provided to 
what groups and who will provj.de it? 

Will this evaluation be used to aid the system 
to improve and extend its internal evaluation 
' capability? 

What is' the internal structure of the budget, 
how will it be monitored? 
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Moral/Ethical/Utllity Queacions 



^Philosophical 

stance : 
JService 

orientation; 
JIvalua tor's 

values: 



Judgments : 



Objectivity: 



Prospects for 
""utility* 

jCost 

^effectiveness) 



Will the evaluation be value free, value based, 
or value plural? 

What social good, if any, will be served by 

this evaluation; whose values will be served? 

Will the evaluator's technical standards and 
his values conflict with the client system's 
and/or sponsor's values; will the evaluator 
face any conflict of interest problems; and 
what will be done about i>ossible conflicts? 

Will the evaluator judge the progr.am; leave 
that up to the client;, or obtain, analyze, 
and report the judgments of Various reference 
groups? 

How will the evaluator avoid being co-opted and 

maintain his objectivity? 
Will the evaluation meet utility criteria of 

relevance, scope, importance, credibility, 

timeliness, and pervasiveness? 
Compared to its potential payoff, will the 

evaluation be carried out at a reasonable 

cost? 



Used with the permission of Daniel L. Stuf flebeam— f rem Meta Evaluation, 
Paper #3, Occasional Paper Series. Kalamazoo, Michigan: The 
Evaluation Center, Western Michigan University, 1975, 
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